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The  Role  of  Temporal  Overlap  of  Visual  and  Auditory  Material 
In  Forming  Dual  Media  Associations 


Abstract 

This  paper  contains  some  results  about  forming  associations  between  the  visual 
and  spoken  material  In  a  dual  media  presentation  such  as  film  or  television.  In 
particular.  It  reports  the  role  played  by  the  overlap  In  time  of  visual  and  auditory 
linguistic  material.  Fourteen  groups  of  college  students  were  shown  a  30  min  film 
which  Introduces  an  assembly  kit.  Its  pieces,  their  names,  and  some  of  their  uses. 
The  film's  visuals  and  narration  could  be  presented  In  synchrony,  or  one  could  be 
shifted  relative  to  the  other  up  to  21  sec.  Subjects  saw  the  film  in  one  of  seven 
versions:  visuals  moved  relative  to  narration  by  -21,  -14,  -7,  0  (synchrony),  7, 

14,  or  21  sec.  They  were  tested  Immediately  or  after  a  week  for  recall  of  the 
names,  given  the  pieces.  Scores  were  highest  at  zero-  and  7-day  delay  for  two 
groups:  synchrony,  and  visuals  7  sec  before  narration.  At  0-delay,  each  of  the 
other  five  groups  scored  about  80%  of  the  highest  groups.  At  7-day  delay,  the  other 
five  groups  scored  differently:  the  3  narration  first  groups  were  about  30%  less 
than  the  two  visuals  first  groups.  The  temporal  order  In  which  visual  and  auditory 
elements  are  presented  differentially  Influences  the  formation  of  dual  media 
associations.  When  auditory  precedes  visuals,  much  of  the  auditory  component  is 
lost.  Two  possible  theoretical  Interpretations  are  given,  and  practical 
applications  are  discussed. 
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The  Role  of  Temporal  Overlap  of  Visual  and  Auditory  Material 
In  Forming  Dual  Media  Associations 

In  many  dual  media  presentations  such  as  film  and  television,  human  speech  Is 
used  In  the  sound  track  to  label  or  describe  or  explain  or  comment  In  some  way  on 
the  visual  Images.  When  the  speaker  Is  not  seen  In  the  visual  images,  such 
narration  is  called  voice  over  (Jacobs,  1970;  Monaco,  1977).  Educational  and 
Instructional  films,  documentaries,  and  television  commercials  frequently  have  sound 
tracks  whose  speech  consists,  partly  or  entirely,  of  voice  over  narration.  It  is 
simpler  to  prepare  than  lip-sync  narration  (synchronization  between  the  movement  of 
the  mouth  of  a  person  In  the  visual  Image  and  the  words  on  the  sound  track),  and  it 
can  be  created  after  the  visual  Images  are  filmed,  allowing  for  flexibility  in 
verbal  content. 

This  paper  reports  results  concerning  the  role  of  the  overlap  in  time  of  the 
visual  and  voice  over  narration  material  In  forming  dual  media  associations. 
(Henceforth,  the  term  narration  will  be  used  to  mean  voice  over  narration,  and  the 
term  visuals  will  be  used  to  mean  the  visual  Images  of  a  film  or  videotape.)  In  this 
study,  the  narration  and  the  visuals  to  which  the  narration  corresponds  are  shifted 
In  time.  For  some  presentations,  the  visuals  come  early;  for  others,  the  visuals 
and  corresponding  narration  come  simultaneously;  and  for  others,  the  narration 
comes  early. 

We  are  testing  one  aspect  of  comprehension,  the  creation  of  associations 
between  visuals  and  narration,  and  In  particular,  the  association  of  an  object  with 
its  name.  We  assess  whether  associations  have  been  formed  by  testing  In  one 
direction  only.  The  cue  is  visual  (an  object),  and  recall  of  the  verbal  part  (the 
object's  name)  is  measured. 
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The  experiment  presented  here  does  not  show  how  people  In  the  various  groups 
with  visuals  and  narration  shifted  in  time  would  perform  on  some  other  tasks.  For 
example,  it  does  not  test  how  well  they  would  answer  questions  based  on  the  film  or 
give  summaries  of  it,  or  perform  tasks  shown  in  the  film.  Our  focus  is  to  establish 
the  role  of  temporal  overlap  In  forming  dual  media  associations. 

Depending  on  the  experimental  outcome,  any  of  several  different  hypotheses 
about  dual  media  processing  may  be  supported  or  rejected.  We  will  briefly  discuss 
four  different  hypotheses  and  the  predictions  they  make. 

A.  First  is  the  dual  code  hypothesis,  whose  most  well  known  proponent  is 
Paivlo  (1971).  It  assumes  that  visual  material  and  verbal  material  are  stored  in 
two  separate  memories.  Internal  associations  between  the  two  are  formed  by  some 
Internal  processing  which  does  not  have  to  occur  at  the  time  of  encoding.  This 
hypothesis  Is  attractive.  It  says  that  people  can  be  presented  visual  and  verbal 
material  at  different  times  and  still  make  very  good  associations.  The  prediction 
from  the  dual  code  hypothesis  Is  that  a  relatively  small  shift  of  visual  and  verbal 
materials  should  not  have  any  Influence  on  the  ability  to  form  associations.  This 
prediction  is  graphed  in  Figure  la. 


Insert  Figure  1  about  here 


B.  The  second  hypothesis  Is  similar  to  A,  in  that  it  assumes  that  there  are 
two  memories  in  which  Information  Is  stored,  one  for  visual  and  the  other  for 
verbal.  It  differs  from  A  In  that  it  assumes  that  associations  between  the  two  are 
formed  at  the  time  of  encoding,  when  the  Information  Is  stored,  that  is,  within  some 
relatively  small  period  of  time. 

Hypothesis  B  has  two  versions.  Hypothesis  B1  assumes  that  visual  and  verbal 
items  are  put  Into  memory  at  the  time  that  they  are  physically  presented.  It 
therefore  predicts  that  the  best  associations  are  formed  with  a  synchronous 
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presentation,  and  that  there  Is  a  uniform  drop  In  both  directions,  when  visual 
precedes  verbal  and  vice  versa. 

Hypothesis  B2  assumes  that  synchrony  in  putting  visual  and  verbal  Items  in 
memory  Is  not  necessarily  equivalent  to  the  physical  presentation  of  the  stimulus. 

In  particular.  It  assumes  that  the  text  consists  of  linguistic  units,  processed  in 
cycles  (as  In  the  theory  of  Kintsch  and  vanDijk,  1978).  Each  linguistic  unit  has  to 
be  divided  Into  propositions,  which  are  linked  together  into  a  coherence  graph. 

Only  then  can  the  conceptual  unit  be  stored.  The  verbal  information  cannot  be  ready 
for  storage,  for  example,  until  a  phrase  is  finished,  and  the  text  base  has  been 
formed.  Associations  from  verbal  and  visual  are  best,  then,  when  the  actual  text 
precedes  the  visuals.  This  causes  the  text  base  to  be  synchronized  with  the 
visuals. 

Thus,  there  are  two  predictions  from  hypothesis  B:  either  (1)  the  best 
associations  are  made  with  visuals  and  narration  in  synchrony;  or  (2)  there  will  be 
some  advantage  when  the  narration  precedes  the  visuals.  These  predictions  are  shown 
in  Figure  lb. 

C.  The  third  hypothesis  assumes  a  single  conceptual  memory  that  operates  as  a 
semantic  network.  (Many  researchers  including  Anderson  (1976),  Collins  and  Quill  Ian 
(1969),  Kintsch  4  vanDijk  (1978),  and  Norman  4  Rumelhart  (1975)  make  this 
assumption.)  Nodes  In  the  network  correspond  to  concepts.  It  is  Important  to  note 
that  the  formulation  here  differs  from  those  of  the  authors  cited  above.  In  that 
concepts  do  not  correspond  to  linguistic  units.  Concepts  can  consist  of  elements 
arising  from  different  media,  for  example,  linguistic  and  pictorial.  When  an 
Instance  of  the  concept  Is  created,  It  will  be  connected  to  the  already  existing 
semantic  network.  (Connections  and  associations  have  the  same  meaning  here.) 

An  Individual  concept  has  some  number  of  associations.  For  example.  If  the 
word  "red"  Is  spoken.  It  can  connect  to,  or  associate  with,  the  visual  concept  of  an 
object  that  Is  red. 
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Examples  of  the  connections  (associations)  made  can  be  seen  when  subjects  are 
naming  some  object.  In  a  previous  study  (Baggett  and  Ehrenfeucht,  In  press)  we 
asked  people  to  name  pieces  In  an  assembly  kit.  The  kit  is  the  same  one  used  In 
this  experiment.  Figure  2  shows  a  piece  from  the  kit  and  Table  1  shows  the  words 
generated,  separated  Into  modifiers  and  nouns.  These  words  are  associated  to  the 
object  shown  in  Figure  2. 


Insert  Figure  2  and  Table  1  about  here 


The  main  assumption  of  this  hypothesis  is  that  visual  concepts  are  able  to  form  more 
associations  than  verbal  concepts,  which  are  far  more  limited.  Me  refer  to  this 
difference  as  “bushy"  and  "skimpy"  concepts  (Baggett,  note  1).  For  example,  it  is 
possible  that  a  concept  formed  from  a  physical  object  can  be  associated  with  the 
verbal  expressions  "red  block",  "orange  brick",  etc.,  as  well  as  to  other  concepts 
formed  from  physical  objects.  On  the  other  hand,  we  assume  that  a  concept  formed 
from  the  verbal  expression  "red  block"  has  fewer  associative  possibilities.  It  is 
Important  to  remember  that  concepts  are  not  necessarily  linguistic. 

Consider  the  situation  when  one  wants  a  direct  association  between  two 
concepts.  One  concept  Is  more  bushy  (can  form  more  associations)  than  the  other. 

The  proper  order  In  which  to  present  the  two  concepts  is  to  give  the  more  bushy  one 
first,  and  the  less  bushy  one  later.  The  mathematical  formulation  will  not  be  given 
here,  but  the  idea  can  be  Illustrated  with  a  simple  example  (see  Figure  3).  Assume 
the  already 


Insert  Figure  3  about  here 


existing  network  E  will  permit  four  connections  to  be  made,  of  types  1,  2,  3,  and  4. 
Two  concepts  are  coming  In  as  Input  to  E.  Concept  X  is  able  to  form  connections  of 
types  1,  2,  4,  and  5.  Concept  Y  is  able  to  form  only  those  of  types  3  and  5. 


Concept  X  is  thus  more  bushy  than  concept  Y.  We  assume  that  X  and  Y  will  connect  to 
E.  We  want  them  to  connect  to  each  other  as  well.  X  and  Y  can  be  connected  only  by 
joining  5  with  5.  What  Is  the  chance  of  this  connection,  depending  on  which  (X  or 
Y)  comes  first? 

We  assume  that  If  connections  are  possible,  they  are  done  with  the  same 
probability,  and  only  one  connection  Is  needed.  Suppose  X  (more  bushy)  comes  first. 
This  situation  Is  shown  in  Figure  3(1).  By  assumption,  it  connects  with  the 
existing  network,  by  any  one  of  1,  2,  and  4.  Next  comes  Y  (less  bushy).  By 
assumption,  it  must  also  connect  to  the  existing  network,  which  now  includes  X.  It 
will  connect  by  either  3  or  5,  each  with  a  50%  probability.  It  can  connect  to  X 
only  by  5,  and  to  E  excluding  X  only  by  3.  Therefore,  the  probability  that  Y  will 
connect  to  X  is  50%.  This  is  the  outcome  when  the  more  bushy  concept  is  presented 
first.  It  is  illustrated  on  the  upper  right  of  Figure  3. 

Now  suppose  Y,  the  less  bushy  concept,  comes  to  the  existing  network  first. 

This  situation  is  shown  in  Figure  3  (II).  Y  connects  by  3.  Next  comes  X,  with  four 
possibilities  (1,  2,  4,  and  5)  of  connecting  with  E.  Only  one  of  these  (5)  is  a 
connection  with  Y.  Therefore,  X's  chance  to  connect  to  Y  is  25%.  This  is  shown  on 
the  lower  right  of  Figure  3. 

This  example  illustrates  that  the  chances  of  creating  a  compound  concept  whose 
parts  are  presented  in  two  modalities  depends  strongly  on  the  order  in  which  the 
parts  are  presented.  Hypothesis  Cl  predicts  that  verbal  material  (skimpy  concepts) 
following  visual  (bushy  concepts)  is  best.  Hypothesis  C2  makes  a  slightly  different 
prediction:  Possibly,  processing  linguistic  material  is  slightly  delayed  relative 
to  visual  (as  in  the  argument  in  B2  above),  so  that  either  delaying  verbal  material 
slightly,  or  presenting  it  in  synchrony  with  visual,  leads  to  the  best  connections. 
These  two  predictions  are  graphed  in  Figure  1(c). 

D.  The  fourth  and  final  hypothetical  framework  gives  the  same  predictions  as 
the  previous  one,  but  is  based  on  a  different  model.*  Paivio  (1971)  can  be  given 
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credit  for  this  one,  as  well  as  Kosslyn  (1980).  Linguistic  material,  according  to 
this  framework,  automatically  creates  visual  images.  The  visual  Images  created  are 
based  on  previous  visual  inputs.  When  verbal  material  follows  visual,  the  verbal 
material  becomes  associated  with  the  just  previously  presented  visual  image.  When 
verbal  material  precedes  visual,  the  visual  images  that  it  associates  with  (call 
them  V)  are  not  necessarily  related  to  the  coming  visual  image  (call  it  VI).  The 
visual  images  automatically  associated  (V)  make  the  correct  association  to  VI 
difficult.  This  framework  predicts,  as  did  C,  that  visuals  and  narration  in 
synchrony,  or  narration  following  visuals,  will  cause  the  best  associations  to  be 
formed. 

Four  different  hypotheses,  and  the  predictions  they  make  about  dual  media 
processing,  have  been  discussed.  Summarizing,  they  are: 

A.  Dual  code  memory,  with  associations  made  at  any  time.  Prediction:  performance 
will  not  vary  with  the  temporal  shift. 

B.  Dual  code  memory,  with  associations  made  at  the  time  of  processing.  Prediction: 
performance  will  be  best  when  visuals  and  narration  are  in  synchrony,  or  when 
narration  precedes  visuals. 

C.  Bushy  and  skimpy  concepts.  Prediction:  performance  will  be  best  when  visuals 
and  narration  are  in  synchrony,  or  when  visuals  precede  narration. 

D.  Mental  imagery.  Prediction:  same  as  (C). 

Method 

Subjects.  Three  hundred  thirty- six  students  in  Introductory  Psychology  at  the 
University  of  Colorado  participated  as  part  of  a  course  requirement.  They  were 
randomly  assigned,  in  groups  of  one  to  five,  to  one  of  14  experimental  conditions, 
with  24  subjects  per  condition.  Half  of  the  conditions  consisted  of  film 
presentations  at  one  of  seven  temporal  overlaps,  with  subjects  tested  immediately 
afterwards.  The  other  half  consisted  of  the  same  temporal  overlaps,  with  subjects 
tested  one  week  later. 
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Design.  The  experiment  was  a  7  x  2  between  subjects  design.  There  were  seven 
types  of  temporal  overlap  (visuals  21,  14,  and  7  sec  before  narration;  visuals  and 
narration  in  synchrony;  visuals  7,  14,  and  21  sec  after  narration),  and  two  delays 
(0-  and  7-day). 

Stimulus  Materials.  The  film,  a  30  min  color  presentation,  was  shot,  edited, 
and  narrated  by  a  professional  film  maker,  James  Otis,  using  silent  super  8  film. 

It  featured  an  assembly  kit,  Fischer  Technik  50,  which  is  similar  to  Lego.  The  kit, 

manufactured  in  Germany,  has  120  total  and  48  different  plastic,  metal,  and  rubber 

2  2 

pieces.  The  smallest  piece  is  5mm  (.2  in  )  and  the  largest,  shown  in  Figure  2,  is 
90  x  45  mm  (3.54  in  x  1.77  in).  The  manufacturers  recommend  its  use  by  children  as 
young  as  six  through  adults. 

Names  to  be  used  in  the  narration  for  the  48  different  pieces  were  selected 
using  a  method  described  in  Baggett  and  Ehrenfeucht  (in  press).  The  names  are 
simple,  short,  easily  matched  with  their  physical  referents,  and  fairly  well 
recalled.  They  also  form  categories.  In  general,  a  name  for  a  piece  consisted  of  a 
noun  (the  category  name)  and  an  adjective  modifier,  to  differentiate  the  piece  from 
other  members  of  the  category. 

The  film  presents  the  48  pieces  in  six  different  scenes,  corresponding  to  six 

categories  of  pieces.  These  are:  4  blocks,  9  joints  and  rails,  6  platforms  and 

plates,  8  rods,  10  wheels,  gears,  and  other  round  pieces,  and  11  special  pieces. 

The  names  are  given  In  Appendix  1.  The  film  has  1626  words  of  narration.  The 

narration  describes  the  pieces,  gives  their  names  and  names  of  their  features,  and 

describes  some  of  their  uses.  The  first  words  of  narration  are  given  in  Appendix  2. 

Using  a  specially  constructed  electronic  arrangement,  simply  by  setting  a 
timer,  we  could  shift  the  narration  and  visuals  with  respect  to  one  another.  We 
selected  seven  such  shifts:  visuals  21,  14,  and  7  sec  before  narration;  visuals 
and  narration  in  synchrony;  and  visuals  7,  14,  and  21  sec  after  narration. 
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The  narration  and  visuals  were  the  same  for  all  14  conditions.  All  that  varied 
was  the  overlap  in  time  of  corresponding  elements.  Expressions  such  as  "This  is..." 
or  "What  you  see  here..."  were  avoided  in  the  narration. 

In  the  extreme  conditions  (visuals  21  sec  before  or  after  the  appropriate 
narration)  the  viewer  could  occasionally  be  hearing  about  piece  X  while  seeing  piece 
Y. 

Procedure .  Subjects  were  run  in  small  groups.  The  instructions  were:  "Today 
you  are  going  to  see  a  film  which  introduces  you  to  an  assembly  kit  of  pieces,  their 
names,  and  some  of  their  uses.  The  kit  is  similar  to  Lego.  After  the  film,  which 
lasts  about  1/2  hour,  you  will  be  given  a  task  to  do,  based  on  what  you  have  seen 
and  heard.  So  just  pay  attention  to  the  film."  (The  people  in  the  7-day  delay 
groups  were  told  to  come  back  one  week  later  to  do  the  task.)  Subjects  were  not  told 
the  exact  nature  of  the  task. 

After  the  film,  and  at  the  appropriate  delay,  each  subject  was  given  a 
collection  of  the  48  actual  pieces  from  the  kit,  a  sheet  with  48  numbered  blanks, 
and  a  folder  with  48  colored  photos  of  the  pieces.  The  photos  were  numbered  l  to 
48,  to  correspond  to  the  blanks.  The  subject  was  instructed  to  write  in  blank  one, 
the  name  of  the  piece  shown  in  photo  one,  etc.  If  the  subject  had  any  doubt  as  to 
which  of  the  48  pieces  was  pictured  in  a  photo,  he  or  she  was  instructed  to  ask  the 
experimenter,  who  would  show  him  or  her  the  actual  piece.  The  subject  was  further 
instructed  to  write  the  names  exactly  as  they  had  been  given  in  the  film. 

Subjects  were  told  there  was  no  penalty  for  guessing.  They  finished  the  task 
in  15  to  30  min.  Before  leaving,  they  filled  out  a  questionnaire  indicating  whether 
they  had  played  with  the  kit  before. 

Results  and  Discussion 


The  data  are  from  336  subjects.  None  of  them  was  familiar  with  the  Fischer 
Technik  assembly  kit,  according  to  answers  on  their  questionnaires.  The  dependent 


measure  was  percentage  correct  on  recall  of  the  names,  using  the  pieces  as  cues.  A 
strict  scoring  criterion  was  used:  Any  deviation  from  exactly  correct  was  scored  as 
wrong. 

Table  2  presents  percentages  correct  and  standard  errors  (in  parentheses)  for 
each  of  14  groups.  Figure  4  shows  the  results  graphically.  Results  of  statistical 
tests  are  given  in  Table  3. 


Insert  Tables  2  and  3  and  Figure  4  about  here 


Two  groups  scored  equally  high  at  zero  delay:  in  synchrony  (S)  and  visuals  7 
sec  before  narration  (V+7).  Their  means  are  45.57  and  44.09%  respectively.  Also, 
two  groups  scored  equally  high  after  a  7  day  delay:  In  synchrony  (S)  and  visuals  7 
sec  before  narration.  Their  means  are  30.12  and  29.94%.  (See  test  4  in  Table  3.) 

At  zero  delay,  the  five  other  groups  scored  significantly  lower  than  S  and  V+7. 
(Test  5  in  Table  3.)  But  scores  in  the  five  groups  were  not  statistically  different 
from  one  another.  (Tests  2  and  6  in  Table  3.)  Thus,  at  zero  delay,  subjects  recall 
the  pieces'  names  best  in  S  and  V+7.  In  all  other  groups  they  are  able  to  recall 
some  names,  about  80%  as  well  (35.94/44.85)  as  In  S  and  V+7. 

The  story  is  different  after  a  week.  As  at  zero  delay,  in  synchrony  (S)  and 
visuals  7  sec  before  narration  are  best.  The  other  five  groups  are  significantly 
lower.  However,  the  five  groups  are  not  identical,  as  they  were  at  zero  delay.  The 
three  narration  first  groups  (narration  21,  14,  and  7  sec  before  visuals)  average 
only  19.8%  correct.  The  two  visuals  first  groups  (visuals  21  and  14  sec  before 
narration)  average  25.4%,  over  28%  higher.  The  three  narration  first  groups  score 
significantly  lower  than  two  visuals  first  groups.  (Test  3  in  Table  3.) 

We  operationally  define  a  measure  of  retention  to  be  the  ratio  of  the 
percentage  correct  after  a  week  to  the  percentage  correct  at  zero  delay.  The 
average  retention  for  the  three  narration  first  conditions  is  55.4%.  For  visuals  21 
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and  14  sec  before  narration,  retention  Is  70.9%.  For  the  conditions,  In  synchrony 
and  visuals  7  sec  before  narration,  retention  Is  66.9%.  For  good  retention.  It  Is 
better  to  err  In  the  direction  of  early  visuals  than  of  early  narration. 

Thfe  main  finding  Is  an  asymmetry  In  the  temporal  shift,  as  shown  In  Figure  4. 
When  visual s  precede  narration  by  up  to  7  sec,  recall  Is  as  good  as  when  visuals  and 
narration  are  In  synchrony.  When  narration  precedes  visuals  by  7  sec  or  rr-ore,  much 
of  the  narration  is  lost,  especially  after  a  delay. 

The  experimental  results  reject  hypotheses  A  and  B  in  the  Introduction  about 
the  mechanisms  Involved  In  forming  dual  media  associations.  They  support  hypotheses 
C  (bushy  and  skimpy  concepts)  and  D  (mental  Imagery),  and  do  not  distinguish  between 
them. 

Applications 

If  either  hypothesis  C  or  hypothesis  D  is  not  rejected  by  future  experiments, 
the  practical  application  to  film  makers  of  where  to  put  narration  is  clear.  Each 
hypothesis  (C  and  D)  makes  the  same  claim,  for  a  different  reason:  In  a  dual  media 
presentation,  the  narration  should  follow,  or  be  In  synchrony  with,  the  visual 
image,  and  never  precede  It. 

Theoretically,  the  correct  temporal  overlap  of  visuals  and  narration  is  not 
restricted  only  to  films  with  voice  over  narration.  It  should  hold  as  well  for 
illustrated  lectures,  slide  shows,  written  text  with  pictures,  etc.  The  principle 
Is  simple:  One  should  present  the  visual  part  early,  or  simultaneous  with  the  text. 
It  Is  better  to  be  too  early  with  the  visual  part  than  too  early  with  the  text. 
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Footnotes 

Thanks  go  to  James  Otis  for  shooting,  editing,  and  narrating  the  film  used 
as  stimulus  material,  to  Jim  Geldel  and  Ashley  Godeaux  for  designing  and 
Implementing  the  electronic  arrangement  used  when  the  film  was  projected,  to 
Caroline  Matsumoto  for  making  the  drawing  for  Figure  2,  and  to  Steve  Stewart 
and  Kevin  Carson  for  helping  to  run  subjects.  This  work  was  supported  by 
Office  of  Naval  Research  contract  N00014-78-C-Q433,  and  is  113  of  the 
Institute  of  Cognitive  Science's  Technical  Report  Series, 
lye  thank  Michel  Denis  for  suggesting  this  formulation. 
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Table  1 

Words  generated  by  14  subjects  who  named  the  piece  shown 
in  Figure  2.  The  number  of  subjects  generating  each 
word  is  in  parentheses. 


modifiers 

nouns 

large  (10) 

plate  (6) 

long  (4) 

platform  (2) 

base  (6) 

bar  (1) 

thick  (1) 

floor  (1) 

multipurpose  (1) 

fork  (1) 

perforated  (1) 

panel  (1) 

red  (1) 

waffle  (1) 

with  holes  (1) 

zigzag  (1) 

Wean  Percentage  Correct  on  Recall  of  the  48  Names  of  the  Pieces 


Statistical  Analyses  and  Their  Results 
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Figure  Captions 


Figure  1;  Predictions  of  performance  in  forming  dual  media  associations  made  by 
Four  different  theoretical  frameworks.  See  the  text  for  further  explanation. 

Figure  la;  Prediction  of  performance  from  dual  code  hypothesis,  with  associations 
formed  at  any  time. 

Figure  lb:  Prediction  of  performance  from  dual  code  hypothesis,  with  associations 
formed  at  the  time  of  encoding. 

(1)  items  are  encoded  at  the  time  they  are  physically  presented. 

(2)  linguistic  items  have  to  be  processed  in  cycles  before  the  resulting 
conceptual  unit  is  stored  in  memory. 

Figure  lc:  Prediction  of  performance  from  bushy  versus  skimpy  concepts  hypothesis, 
and  from  mental  imagery  hypothesis. 

Figure  2:  A  piece  from  the  Fischer  Technik  50  assembly  kit.  Its  actual  size  is 
90  x  45  x  5  mm  (3.54  x  1.77  x  .2  in).  Words  that  subjects  generated  in  naming  the 
piece  are  given  in  Table  1. 

Figure  3:  A  direct  association  is  wanted  between  concept  X  (bushy)  and  concept  Y 
(skimpy).  I  shows  the  situation  when  X  is  presented  to  existing  memory  first.  The 
probability  that  X  and  Y  become  associated  is  .5.  II  shows  the  situation  when  Y 
is  presented  to  existing  memory  first.  The  probability  that  X  and  Y  become  associated 
is  .25.  Therefore,  to  increase  the  probability  that  X  and  Y  are  associated,  the  proper 
order  is  to  present  X  first  and  Y  later. 

Figure  4:  The  results  in  Table  2  presented  in  graph  form.  Mean  percentage  correct 
on  recall ,  as  a  function  of  temporal  shift  of  narration  and  visuals,  and  delay  (zero 
or  7-day)  between  film  and  test. 
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Appendix  I 

Names  for  the  48  pieces  in  the  assembly  kit,  grouped 
according  to  the  six  scenes  in  the  film 


1.  blocks 
large  block 

large  block  with  holes 
small  block 

small  block  with  red  knob 

2.  joints  and  rails 
cube  joint 
triangle  joint 
slanted  joint 
H-joint 

flat  square  joint 
corner  joint 
joint  with  half  axle 
long  red  rail 
short  red  rail 

3.  platforms  and  plates 

large  platform  (shown  in  Fig.  2) 
small  platform 

narrow  platform  with  half  axles 
large  plate 
medium  plate 
small  plate 


4.  wheels,  gears,  and  other  round  pieces 
smooth  red  wheel 

black  ring 

wheel  with  movable  handle 
tire 

large  black  gear 
small  black  gear 
large  red  gear 
small  red  gear 
large  pulley  wheel 
small  pulley  wheel 

5.  rods 
short  metal  rod 
medium  metal  rod 
long  metal  rod 
extra  long  metal  rod 
short  grey  rod 

long  grey  rod 

grey  rod  with  square  end 

crankshaft 
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Names  for  the  48  pieces  in  the  assembly  kit,  grouped 
according  to  the  six  scenes  in  the  film 


1.  blocks 
large  block 

large  block  with  holes 
small  block 

small  block  with  red  knob 

2.  joints  and  rails 
cube  joint 
triangle  joint 
slanted  joint 
H-joint 

flat  square  joint 
corner  joint 
joint  with  half  axle 
long  red  rail 
short  red  rail 

3.  platforms  and  plates 

large  platform  (shown  in  Fig.  2) 
small  platform 

narrow  platform  with  half  axles 
large  plate 
medium  plate 
small  plate 


4.  wheels,  gears,  and  other  round 
smooth  red  wheel 

black  ring 

wheel  with  movable  handle 
tire 

large  black  gear 
small  black  gear 
large  red  gear 
small  red  gear 
large  pulley  wheel 
small  pulley  wheel 

5.  rods 
short  metal  rod 
medium  metal  rod 
long  metal  rod 
extra  long  metal  rod 
short  grey  rod 

long  grey  rod 

grey  rod  with  square  end 

crankshaft 
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6.  special  pieces 

string 

crank 

hook 

spool 

C-clip 

red  tube 

spring 

rubber  band 

1- hole  hitch 

2- hole  hitch 

3- hole  hitch 
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Appendix  2 

The  opening  narration  to  the  film 

This  film  is  about  an  assembly  kit  that  consists  of  48  different  plastic, 
metal,  and  rubber  pieces.  The  largest  piece  is  the  large  platform,  and  the  smallest 
is  the  C-clip. 

You  can  build  many  objects  from  the  pieces.  .  .  simple  abstract  structures.  .  . 
a  unicorn.  .  .  an  antenna  that  moves  when  you  turn  the  crank.  .  .  a  little  car.  .  . 
a  dump  truck.  .  .  that  dumps!.  .  .  or  even  a  fanciful  antennaed  creature  with 
plates.  This  film  will  introduce  you  to  the  different  pieces,  their  names,  and 
some  of  their  uses. 

The  first  category  is  blocks.  There  are  four  blocks.  The  first  one  is 
called  the  large  block.  It  has  a  knob  on  one  side  and  grooves  on  its  other  five 
sides. 

An  interesting  object  can  be  built  using  only  four  large  blocks.  The  secret 
is  to  build  two  separate  L-shapes,  each  with  two  large  blocks,  and  then  to  join 
them.  The  large  block. 

The  next  one  is  the  large  block  with  holes.  It  is  identical  to  the  large 
block,  except  that  there  are  two  holes  through  its  center. 

The  third  block  is  the  small  block.  Like  all  blocks,  it  is  grey,  with  a 
knob  on  one  side  and  grooves  on  its  other  five  sides. 
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